Parallel Multistage Wide Neural Network

Deep learning networks have achieved great success in many areas such as in large scale image processing. They usually need large computing resources and time, and process easy and hard samples inefficiently in the same way. Another undesirable problem is that the network generally needs to be retrained to learn new incoming data. Efforts have been made to reduce the computing resources and realize incremental learning by adjusting architectures, such as scalable effort classifiers, multi-grained cascade forest (gc forest), conditional deep learning (CDL), tree CNN, decision tree structure with knowledge transfer (ERDK), forest of decision trees with RBF networks and knowledge transfer (FDRK). In this paper, a parallel multistage wide neural network (PMWNN) is presented. It is composed of multiple stages to classify different parts of data. First, a wide radial basis function (WRBF) network is designed to learn features efficiently in the wide direction. It can work on both vector and image instances, and be trained fast in one epoch using subsampling and least squares (LS). Secondly, successive stages of WRBF networks are combined to make up the PMWNN. Each stage focuses on the misclassified samples of the previous stage. It can stop growing at an early stage, and a stage can be added incrementally when new training data is acquired. Finally, the stages of the PMWNN can be tested in parallel, thus speeding up the testing process. To sum up, the proposed PMWNN network has the advantages of (1) fast training, (2) optimized computing resources, (3) incremental learning, and (4) parallel testing with stages. The experimental results with the MNIST, a number of large hyperspectral remote sensing data, CVL single digits, SVHN datasets, and audio signal datasets show that the WRBF and PMWNN have the competitive accuracy compared to learning models such as stacked auto encoders, deep belief nets, SVM, MLP, LeNet-5, RBF network, recently proposed CDL, broad learning, gc forest etc. In fact, the PMWNN has often the best classification performance

Uzaktan algılama görüntülerinin sınıflandırılması için sınır özniteliklerinin belirlenmesi ve adaptasyonu algoritması

Various types of sensors collect very large amounts of data from the earth surface. The characteristics of the data are related to sensor type with its own imaging geometry. Consequently, sensor types affect processing techniques used in remote sensing. In general, image processing techniques used in remote sensing are usually valid for multispectral data which is relatively in a low dimensional feature space. Therefore, advanced algorithms are needed for hyperspectral data which have at least 100-200 features (attributes/bands). Additionally, the training process is very important and affects the generalization capability of a classifier in supervised learning. Enough number of training samples is required to make proper classification. In remote sensing, collecting training samples is difficult and costly. Consequently, a limited number of training samples is often available in practice. Conventional statistical classifiers assume that the data have a specific distribution. For real world data, these kinds of assumptions may not be valid. Additionally, proper parameter estimation is difficult, especially for hyperspectral data. Normally, when the number of bands used in the classification process increases, precise detailed class determination is expected. For high dimensional feature space, when a new feature is added to the data, classification error decreases, but at the same time, the bias of the classification error increases. If the increment of the bias of the classification error is more than the reduction in classification error, then the use of the additional feature decreases the performance of the decision algorithm. This phenomenon is called the Hughes effect, and it may be much more harmful with hyperspectral data than with multispectral data. Our motivation in this study is to overcome some of these general classification problems by developing a classification algorithm which is directly based on the available training data rather than on the underlying statistical data distribution. Our proposed algorithm, Border Feature Detection and Adaptation (BFDA), uses border feature vectors near the decision boundaries which are adapted to make a precise partitioning in the feature space by using maximum margin principle. The BFDA algorithm well suited for classification of remote sensing images is developed with a new approach to choosing and adapting border feature vectors with the training data. This approach is especially effective when the information source has a limited amount of data samples, and the distribution of the data is not necessarily Gaussian. Training samples closer to class borders are more prone to generate misclassification, and therefore are significant feature vectors to be used to reduce classification errors. The proposed classification algorithm searches for such error-causing training samples in a special way, and adapts them to generate border feature vectors to be used as labeled feature vectors for classification. The BFDA algorithm can be considered in two parts. The first part of the algorithm consists of defining initial border feature vectors using class centers and misclassified training vectors. With this approach, a manageable number of border feature vectors is achieved. The second part of the algorithm is adaptation of border feature vectors by using a technique which has some similarity with the learning vector quantization (LVQ) algorithm. In this adaptation process, the border feature vectors are adaptively modified to support proper distances between them and the class centers, and to increase the margins between neighboring border features with different class labels. The class centers are also adapted during this process. Subsequent classification is based on labeled border feature vectors and class centers. With this approach, a proper number of feature vectors for each class is generated by the algorithm. In supervised learning, the training process should be unbiased to reach more accurate results in testing. In the BFDA, accuracy is related to the initialization of the border feature vectors and the input ordering of the training samples. These dependencies make the classifier a biased decision maker. Consensus strategy can be applied with cross validation to reduce these dependencies. In this study, major performance analysis and comparisons were made by using the AVIRIS data. Using the BFDA, we obtained satisfactory results with both multispectral and hyperspectal data sets. The BFDA is also a robust algorithm with the Hughes effect. Additionally, rare class members are more accurately classified by the BFDA as compared to conventional statistical methods.  Keywords: Remote sensing, hyperspectral data classification, consensual classification.Geleneksel görüntü işleme tekniklerinin direkt olarak uzaktan algılamaya uygulanması, sadece multispektral datalar için geçerli olabilir. Öznitelik vektörü boyutu 100-200 civarında olan hiperspektral dataların analizi için gelişmiş algoritmalara ihtiyaç vardır. Bununla birlikte, uzaktan algılamada, genellikle sınırlı sayıda eğitim örneğinin olması, özellikle öznitelik vektörünün boyutunun büyük olduğu hiperspektral datalarda, parametrik sınıflayıcıların kullanımını kısıtlar. Bu çalışmanın amacı, istatistiksel dağılıma bağlı olmayan, sadece eldeki eğitim örneklerine dayanan bir algoritma geliştirerek yukarıda özetlenen uzaktan algılama için genel sınıflandırma problemlerinin üstesinden gelmektir. Önerilen Sınır Özniteliklerinin Belirlenmesi ve Adaptasyonu (SÖBA) algoritması, karar yüzeylerine yakın sınır öznitelik vektörlerini kullanır ve bu sınır öznitelik vektörleri, maksimum marjin prensibini sağlayacak şekilde adapte edilerek, öznitelik uzayında doğru bölütlemenin yapılmasını sağlar. SÖBA algoritması iki bölümden oluşur. İlk aşamada sınır öznitelik vektörlerinin başlangıç değerleri uygun eğitim kümesi elemanlarından, yönetilebilir sayıda atanır. Daha sonra uygulanan adaptasyon işlemiyle, öğrenme süreci gerçekleştirilerek sınır özniteliklerinin, sonuç değerlerine ulaşması hedeflenir. Sınıflandırma sonuç sınır öznitelik vektörlerine olan  en yakın 1 komşuluk (1-EK) kuralı uyarınca yapılır. Ek olarak, SÖBA algoritmasının sınır öznitelik vektörlerinin başlangıç değerlerine ve eğitim kümesi elemanlarının eğitimde kullanılma sırasına bağlı olarak her çalışmasında kabul edilebilir derecede farklı sınır karar yüzeyleri oluşturması, konsensüs yapılarda kullanılması için elverişli bir özelliktir. Böylece birçok defa çalıştırılan SÖBA kararlarının uygun kurallarla birleştirilmesiyle tek bir sınıflayıcının aldığı karardan çok daha doğru kararlar elde edilebilir. Anahtar Kelimeler: Uzaktan algılama, hiperspektral data sınıflandırma, konsensüs

2K09 and thereafter : the coming era of integrative bioinformatics, systems biology and intelligent computing for functional genomics and personalized medicine research

Significant interest exists in establishing synergistic research in bioinformatics, systems biology and intelligent computing. Supported by the United States National Science Foundation (NSF), International Society of Intelligent Biological Medicine (http://www.ISIBM.org), International Journal of Computational Biology and Drug Design (IJCBDD) and International Journal of Functional Informatics and Personalized Medicine, the ISIBM International Joint Conferences on Bioinformatics, Systems Biology and Intelligent Computing (ISIBM IJCBS 2009) attracted more than 300 papers and 400 researchers and medical doctors world-wide. It was the only inter/multidisciplinary conference aimed to promote synergistic research and education in bioinformatics, systems biology and intelligent computing. The conference committee was very grateful for the valuable advice and suggestions from honorary chairs, steering committee members and scientific leaders including Dr. Michael S. Waterman (USC, Member of United States National Academy of Sciences), Dr. Chih-Ming Ho (UCLA, Member of United States National Academy of Engineering and Academician of Academia Sinica), Dr. Wing H. Wong (Stanford, Member of United States National Academy of Sciences), Dr. Ruzena Bajcsy (UC Berkeley, Member of United States National Academy of Engineering and Member of United States Institute of Medicine of the National Academies), Dr. Mary Qu Yang (United States National Institutes of Health and Oak Ridge, DOE), Dr. Andrzej Niemierko (Harvard), Dr. A. Keith Dunker (Indiana), Dr. Brian D. Athey (Michigan), Dr. Weida Tong (FDA, United States Department of Health and Human Services), Dr. Cathy H. Wu (Georgetown), Dr. Dong Xu (Missouri), Drs. Arif Ghafoor and Okan K Ersoy (Purdue), Dr. Mark Borodovsky (Georgia Tech, President of ISIBM), Dr. Hamid R. Arabnia (UGA, Vice-President of ISIBM), and other scientific leaders. The committee presented the 2009 ISIBM Outstanding Achievement Awards to Dr. Joydeep Ghosh (UT Austin), Dr. Aidong Zhang (Buffalo) and Dr. Zhi-Hua Zhou (Nanjing) for their significant contributions to the field of intelligent biological medicine

DFT/RDFT Filter Banks with Symmetric Zero-Phase Nonoverlapping Analysis/Synthesis Filters and Applications

Digital filter banks satisfying perfect reconstruction (PR) condition are very useful in many applications. In this report, new filter bank structures related to the complex and real discrete Fourier transforms (DFT and RDFT) are introduced, and their performance in applications such as image fusion are investigated as compared to wavelets. The importance of zero-phase filter banks has been increasing since they can be effectively used within filter banks without time shift (or space shift in 2-D). The art of designing zero-phase low pass and high pass analysis filters is well established. There is also no phase distortion occurring within the filter banks. For a real input signal, analysis/synthesis banks with such filtering always give a real output signal. Consequently, the proposed DFT/RDFT filter banks with symmetric zero-phase analysis filters may be suitable for a wide range of applications in signal and image processing. The method developed was used in 1-D, and 2-D subband decomposition tasks. Image fusion was especially the application studied in detail. In terms of performance, the results with the new method was better than the results obtained with the wavelet approach using Daubechies 1 (Haar) wavelet in all the applications comparatively studied

Probabilistic Matching Pursuit for Compressive Sensing

Compressive sensing investigates the recovery of a signal that can be sparsely represented in an orthonormal basis or overcomplete dictionary given a small number of linear combinations of the signal. We present a novel matching pursuit algorithm that uses the measurements to probabilistically select a subset of bases that is likely to contain the true bases constituting the signal. The algorithm is successful in recovering the original signal in cases where deterministic matching pursuit algorithms fail. We also show that exact recovery is possible when the number of nonzero coefficients is upto one less than the number of measurements. This overturns a previously held assumption in compressive sensing research

PARALLEL, PROBABILISTIC, SELF-ORGANIZING, HIERARCHICAL NEURAL NETWORKS

A new neural network architecture called the Parallel Probabilistic Self-organizing Hierarchical Neural Network (PPSHNN) is introduced. The PPSHNN is designed to solve complex classification problems, by dividing the input vector space into regions, and by performing classification on those regions. It consists of several modules which operate in a hierarchically during learning and in parallel during testing. Each module has the task of classification for a region of the input information space as well as the task of participating in the formation of these regions through post- and pre-rejection schemes. The decomposition into regions is performed in a manner that makes classification easier on each of h e regions. The post-~jector submodule performs a bitwise statistical analysis and detection of hard to classify vectors. The pre-rejector module accepts only those classes for which the module is trained and rejects others. The PNS module is developed as a variation of the PPSHNN module. If delta rule networks are used to build the submodules of PNS, then it uses piecewise linear boundaries to divide the problem space into regions. The PNS module has a high classification accuracy while it remains relatively inexpensive. The submodules of PNS are fractile in nature, meaning that each such unit may itself consist of a number of PNS modules. The PNS module is discussed as the building block for the synthesis of PPSHNN. . The SIMD version of PPSHNN is implemented on MASPAR with 16k processors. On all the experiments performed, this network has outperformed the previously used networks in terms of accuracy of classification and speed

A Novel Evolutionary Global Optimization Algorithm and its Application in Bioinformatics

Gray Code Optimization (GCO) algorithm is a deterministic algorithm based on the Gray code, binary numbers representation. It sometimes suffers from slow convergence and sub-optimal solutions. Expectation Maximization (EM) algorithm is used to analyze how the GCO explores the search space. The investigation of how the GCO generates a population indicates that it is similar to generating samples with a mixture Gaussian distribution. The EM algorithm extracts a three components mixture Gaussian model. Based on these findings, a novel stochastic optimization algorithm based on the mixture Gaussian model is proposed. The new Mixture Gaussian Optimization (MGO) algorithm is not only a continuous stochastic algorithm, but also provides a rigorous mathematic model for answering some theoretical questions. A proof of the convergence of MGO based on the Markov Model is given. The MGO algorithm is applied to the global optimization problems in bioinformatics. For example, the conformations available to a molecule can have a dramatic effect on its activity. Obtaining global minimum energy conformations of molecule is a very hard optimization problem. The difficulty arises from the following two factors: the conformational space of a reasonable size molecular is very large, and there are many local minima that are hard to sample efficiently. The energy landscape in the conformational space is very rugged, and there are many large barriers between local minima. In this report, the MGO algorithm is used to search the conformation space and locate the global minimal energy structure

A PARALLEL IMPLEMENTATION OF BACKPROPAGATION NEURAL NETWORK ON MASPAR MP-1

One of the major issues in using artificial neural networks is reducing the training and the testing times. Parallel processing is the most efficient approach for this purpose. In this paper, we explore the parallel implementation of the backpropagation algorithm with and without hidden layers [4][5] on MasPar MP-I. This implementation is based on the SIMD architecture, and uses a backpropagation model which is more exact theoretically than the serial backpropagation model. This results in a smoother convergence to the solution. Most importantly, the processing time is reduced both theoretically and experimentally by the order of 3000, due to architectural and data parallelism of the backpropagation algorithm. This allows large-scale simulation of neural networks in near real-time

DENSE WAVELENGTH DIVISION MULTIPLEXING/DEMULTIPLEXING BY THE METHOD OF IRREGULARLY SAMPLED ZERO CROSSINGS

The tremendous growth of Internet traffic has created increasing demand on high capacity optical communications networks. Dense wavelength division multiplexing (DWDM) networks has emerged as a very attractive option. Systems for optical wavelength demultiplexing have generated much interest in research and development. Since its first appearance, the Arrayed Waveguide Grating (AWG) has become the mainstay for the high channel count demultiplexing applications. In classical AWG design, the number of wavelength channels to be resolved is directly limited by the Free Spectral Range (FSR), and ultimately the channel count of an AWG is limited by the number of arrayed waveguides that could be fit on the substrate on which the AWG is fabricated. The harmonic images generated outside the FSR limit additional channels beyond FSR. In this work, a novel array waveguide grating (AWG) design method is proposed to achieve large channel count in a single stage. The Method of Irregularly Sampled Zero Crossings (MISZC) with a spherical wave applies non-periodic/irregular placement of apertures of the grating structure combined with holographic techniques based on zero crossings with a virtual spherical reference wave. As a result, the restriction of free spectral range (FSR) for the regular AWG is eliminated. Thus, high channel count (\u3e500) can be achieved in a single stage with reasonable noise level. Theoretical derivation and analysis of MISZC is presented. Detailed simulations using BeamPROP™ and Matlab™ tools are reported to show good agreement with the theory and analysis

PNS MODULES FOR THE SYNTHESIS OF PARALLEL SELF-ORGANIZING HIERARCHICAL NEURAL NETWORKS

The PNS module is discussed as the building block for the synthesis of parallel, selforganizing, hierarchical, neural networks (PSHNN). The PNS consists of a prerejector (P-unit), a neural network (N-unit) and a statistical analysis unit (S-unit). The last two units together are also referred to as the NS unit. The P- and NS-units are fractile in nature, meaning that each such unit may itself consist of a number of parallel PNS modules. Through a mechanism of statistical acceptance or rejection of input vectors for classification, the sample space is divided into a number of subspaces. The input vectors belonging to each subspace are classified by a dedicated set of PNS modules. This strategy results in considerably higher accuracy of classification and better generalization as compared to previous neural network models. If the delta rule network is used to generate the N-unit, each subspace approximates a linearly separable space. In this sense, the total system becomes similar to a piecewise linear model